2011/Notes/AgileFLOSS in-person inclusion
Agile/FLOSS discussion, Community Leadership Summit 2011Sumana Harihareswara (@brainwane) & David Strauss See below for considerations/conclusions * Coordinating software change with the community - ??? - Any successful, long-lived project has to cope with change, and such change can't entirely come from universal community consensus. Working groups tend to form -- but they often coordinate and work offline (with "sprints") or by email. This work can't exist forever in that vaccuum, so the group presents progress back to the community. Such "progress updates" often spark confrontation. The working groups get frustrated when community stakeholders who feel left out of the process criticize details of the group's work. This interaction is demotivating to the people in the working groups and the community members. Many community members see the working groups as exclusionary, especially because not every stakeholder has the means or time to participate in them. Is there a better way to have stakeholders and working group participants plan and interact, before, during, or after "sprints"? (Posted here by David Strauss, a contributor to the Drupal project) *: Would love to know how people have made this fit with Agile. - Sumana Harihareswara *: This is an area that I have an interest in (my community group has recently gained a seat on the Java Community Process committee and we're working on this exact problem with Java standards work) ''- ''Martijn Verburg *: At WordPress our working groups do all communication publicly on group blogs or in IRC, so the broader community doesn't have much room to resent them, since getting involved is as easy as leaving a blog comment. We have bigger challenges with keeping a strong connection between the core team and the working groups (mostly failing there right now), and with community resentment of the core team over various decisions because some (vocal) community members don't think the core team gives community opinions (in many cases single opinions) the recognition/weight they deserve. - Jane Wells * David notes:this sort of thing -> bikeshedding, demotivation. Sometimes people take vacation & money to go to these things, then feel attacked by people who weren't there!community feels not included as a stakeholder. Some sprints are publicly announced, other people decide to come. some are more invite-only Transparency is one question. "Here's what we're going to do...." Often, initiative is organized well ahead of a sprint. Some start with tons of prior planning, architecture. Others are more ... initiative is necessary, needs to get done, such as configuration management for Drupal 8 At sprint, we hammered out how we wanted to build it, built prototype. Lots of little decisions got made. what serialization format? YaML/JSON/etc. Audrey: When starting calagator, we we doing both agile sprints and time-boxed in-person sprints at the same time. Index cards with as short as 45-minute iterations. Sumana, at a recent code sprint in Berlin, worked on trying to get participation from remote participants by making sure that people talked about the things they were doing in person on IRC and etherpad, but this required two people working full-time on this documentation effort.(-- I may have missed something here) Etherpadding 3 days straight... When there is a decision to be made at an in-person sprint where it's more important to pick something and go with it than waiting for public comment. Afterwards, people who weren't participating in the sprint assume that the decision was made without any thought. General lack of good faith. Doing everything as a working document in etherpad avoids needing post-sprint documentation because the process is being documented in real-time Does resistance come from social reasons? technical reasons? things that can't be easily swapped out for each other? details get picked outliving doc on Etherpad will inherently contain all the details! Paul: we work with Rackspace on OpenStack Design sprints every 6 months we tell people way ahead of time We tell the world what our product roadmap 6 months ahead of time, ask for feedback we work in Agile inside our org, decided Agile doesn't work in FLOSS, some people do not want to work in 2-week iterations.Rackspace: every 3 months, a big release.We have milestones in release. D1, d2, d3, about monthly. Smaller iterations within that. Developer can sign up to target a specific 1- to 3-month window! OpenStack: breakdown of contributors. Lots of corporations. Not all rackspace Mozilla just started rapid release of project, every 6 months or so, which is the same cadence they used to release security stuff at, which is goodOnce you get beyond a certain speed, it's even more community-friendly. People work at their own pace, and then within 12 weeks their stuff is in the product! Some Drupal frustration: multiyear release cycle for core. "If I don't get my voice in in this initiative, I'm out for the count for 1-2 years wrt this subsystem!" Mozilla: to do rapid release, you have to have:1) automated testing2) lots of test users nightly Aurora etc.to find the bugs.3) date-based releaseswe're using Mercurial. There's enough interoperability you can have bridges from Git <->Mercurial 3 months is the worst of both worlds! -- Mozilla -- not enough stability, maintaining too many branches, but not enough speed either Drupal:(something similar) You want people outside your org deploying your alpha & beta releases. This is easy for software that gets deployed to individual machines with automated updates, but much harder for web applications that have a legacy of only releasing things after they've ben tested in production. Calagator example about 6 users thoroughly using it at any given time, but the main pdx tech site runs all code before it goes into a stable release Some of the participants:*OpenStack *Wikimedia *Calagator *Mozilla *Drupal *Michael Downey, OpenMRS. Weeklong sprint cycles, announced at least a month in advance. Virtual. People sign up to do something during those sprints. Have been doing this since March. Every so often, an off week to do minor misc issues. Team has about 6 fulltime paid devs. Most of participants, in terms of #s, volunteer-based. GSoC, or other folks from larger dev community. Very informal. wiki page for each sprint. Each person's goals. Our project uses Atlassian tools - JIRA, Greenhopper, Confluence. Greenhopper: Agile plugin for JIRA. Kanban or Scrum. Provides cards/rankings/story status/planning. In JIRA: dashboard for each sprint. In terms of code output, # of commits hasn't changed too much, but focus has. Past: people did whatever they wanted. Anecdotal feedback from user community: positive, They hear & see us working on specific aspects of app based on their feedback (surveys). Don't know whether we are more productive in aggregate. This is helpful for momentum, camaraderie for core developers. Setting aside times to do live communication, working on a single highlevel outcome. *: Still exploring how to ... explore new technologies, documentation or UI design sprints, etc. to get more people in the community involved. *: Worked with ThoughtWorks, has donated developers. Agile shop. But OpenMRS uses non-Mingle tool. *: We have a seasonal dev cycle. Big Summer of Code program. That throws off feel of activity; part-time developers mentor fulltime students. So, whether this is changing community contribution -- will be seen later, maybe in a year. Some GSoC students do not take part in sprints. *: Hey Michael - about how many nonpaid developers does OpenMRS regularly have? you mentioned the larger dev community... *GSoC project.... if the first time the community sees a project is at the end of the summer, that'll be hard to integrate into the project. Drupal's having a problem getting that work merged into trunk. sometimes code doesn't get merged, but the idea does. Sort of ends up being a prototype for something that later gets written... Audrey: Idea of decoupling development process from the internal process at the org is interesting. OpenStack: Process started in April. As we go into design summit, we as a community decide on must-have features that we will delay a release for. Security, or integrating fundamental engines into the system (e.g., distributed scheduler).Weekly IRC meetings, community manager sets up, meetbot, everything is logged. So people can see exactly who has action items, etc. Need a strong community leader to ensure everything is organized.We have not always been able to cut something and push release on time.Important to decide process ahead of time so ... not writing policy in heat of the moment Importance of intermediate checkins Software is in a state where people are using unstable released versions decided process in person at the design summit "everyone come with your idea!"etherpadeveryone lays out ideas ahead of time OpenStack does this with Launchpad.Maintaining community:with blueprints. Community leaders publicize blueprints ahead of design summitanyone can write onelink to it from wiki, whiteboardhave discussions and then at the summitunconferencewe go argue it outsometimes we have a resolution, sometimes notusually we arrive at a general "who will do what", see different interests then have conversation in person, document it"this is the next 3 months of work to do"corp perspective sometimes want to get things done early. Break big things as early as possible in the release cycle! Know ahead of time.... listed on wiki page, who's the key people in which subcommunities.Those key people act as PR, ensuring people know about blueprints, design summits ahead of time. OpenStack pushes it to devs to do a lot of communicating Question about impasses & agree-to-disagree. OpenStack: API is an example here. There's a common set of components you can do with the server. There's an extension mechanism. We defined ahead of time a way to go there... Parking lot mechanism for "agree to disagree".... how to box off disagreements so other work can happen? Decentralized Drupal: nearly everything happens not in core, but in subsystems. So there's not as much blocking as in some projects...but since many people contribute in multiple parts, their discussion in one place might be distracting or demoralizing wrt other components, core. projects have different cultures wrt reversions, commit hooks & tests Launchpad's merge proposals, comments system is good, but can slow down process with discussion. OpenStack: sprint perspective, we have allocated time, written code and moved on, but there is leftover work in discussing -- doing the social merge! how do you allocate for this? 40 points? give more points when you think they'll cause more overhead. a pain bump. David suggests: until you have it merged, you don't move on to the next step. It's your job to get it merged, both passing tests & passing code review. "this is an improvement but does not consider x, y, & z" -- short release cycles, can still move forward."this is a step back" -- long release cycle means BIG FIGHT. short relase cycle (and ability to revert changes) means you can experiment. Getting people to try the betas, alphas -- Mozilla has "channels." Trouble is getting people to get onto channel. & server (need high reliability, etc.) vs desktop, easy-to-update/fix issues. "Considerations":*Talk about *everything* at sprints in systems like Etherpad to (1) show the verbose decision-making process and (2) allow people to participate who aren't there. *Use daily blog posts at sprints. *Use short release cycles to reduce urgency for people to push in their work and have their needs handled ASAP (to avoid a very long wait until the next release). *Have a clear process for stakeholders to propose and participate in the planning process for work. Intermediate checkins are good. *Consider having an open door for backing things out. *Have the overall development process be a big tent housing internal and community development, all subject to the same "big tent" rules. This includes milestones, blueprints, etc. *Release in stages to the community: have broad deployment of alpha/beta releases to give new work the acid test (and the satisfaction of distributing someone's good work). *To release rapidly, you need automated testing, date-based releases, and a big pool of testers who'll test alpha & beta versions. *releasing every 3 months is the worst of both worlds. There's not enough stability, and you're maintaining too many branches, but not enough speed either.